resolves dbt-labs/dbt-adapters#527
docs dbt-labs/docs.getdbt.com/#

Problem

By default, we are using a MERGE operation when running an insert_overwrite or microbatch operation. But this is not the most performant solution. In fact, we have different options with their pros and cons. Currently we only rely on merge that is not the best performant solution and it doesn't allow fully parallelism for microbatch operations.

Solution

In this PR we propose adding a new incremental_substrategy for microbatch and insert_overwrite that will allow the user to select the best logic based on their needs.

Additionally, based on experience the best combo I've seen is:

• For static partitions: delete+insert without a transaction to enable full parallelism
• For dynamic partitions: bq copy

For that reason, we can add an additional optimised option that will run the best operation available (and that can be updated in the future if needed)

{{
    config(
        incremental_strategy={
                               "insert_overwrite",
                               "microbatch"
                              },
        incremental_substrategy={
                                 'merge', -- by default if no substrategy is specified
                                 'copy_partitions', -- same as defined in partitions_by
                                 'delete+insert', 
                                 'optimised' -- if partitions is defined (static), used delete+insert, if not (dynamic), use copy partitions
                                 },
       ...
    )
}}

Note that the copy_partitions will also be backwards compatible by checking if it has been defined in the partition_by config block.

We could force every user to use the optimised by default with the new release, but I just feel unease given that merge has been the default for ever and people might have built logics assuming this behaviour...?

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX